System and method for view management in three dimensional space

ABSTRACT

A method for managing a display space for a 3D environment is provided. A 3D scene having at least one scene object ( 105 ) is displayed and the visible surfaces ( 110 ) of the scene objects are represented as visible space in a 2D view plane representation. Controllable objects that are to be placed in the scene are defined by parameters such as size, placement priority, proximity relationships and the like. The available space for placing controllable objects, which can include empty space and low priority background and foreground regions, is determined for each controllable object. The placement ( 125 ) for controllable objects in the 3D space is then determined in accordance with at least placement parameter and one of the visible space and available space of the view-plane representation ( 120 ) such that view management objectives, such as not occluding important scene objects, are accomplished.

CLAIM FOR PRIORITY TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional PatentApplication serial No. 60/291,798, filed on May 17, 2001, entitled ViewManagement For Virtual And Augmented Reality, which is herebyincorporated by reference in its entirety.

STATEMENT OF GOVERNMENT RIGHTS

[0002] The present invention was made in part with support from theNational Library of Medicine, Grant No. 5-R01 LM06593-02 and the Officeof Naval Research, Contract Nos. N00014-99-1-0683, N00014-99-1-0249 andN00014-99-1-0394. Accordingly, the United States government may havecertain rights to this invention.

FIELD OF THE INVENTION

[0003] The present invention relates generally to three dimensionalimaging and more particularly relates to a system and method formanaging the placement of controllable objects in a three dimensionalprojection.

BACKGROUND OF THE INVENTION

[0004] Computer graphics systems which are commonly used today generallyprovide a representation of the workspace, or display screen, occupiedby the various elements of the scene. Designing a graphical userinterface (GUI) for viewing and manipulating a virtual three-dimensional(3D) space requires creating a set of objects and their properties,arranging them in a scene, setting a viewing specification, determininglighting and rendering parameters, and deciding how to update thesedecisions for each frame. Some of these decisions may be fullyconstrained; for example, a simulation may determine the position andshape of certain objects, or the viewing specification may be explicitlycontrolled by the user. In contrast, other decisions must be resolved bythe GUI designer. Of particular interest are those decisions thatdetermine the spatial layout of the projections of objects on the viewplane. These decisions can be referred to collectively as viewmanagement. For example, some objects may be sufficiently important tothe user's task that they should not be occluded. In addition, themembers of a group of related objects may need to be placed together toemphasize their relationship.

[0005] In a static scene, observed from a fixed viewing specification,view-management decisions might be made in advance, by hand, and holdthroughout the life of an application. It is also common in both 2D and3D interactive GUIs to avoid automating view management when possible.For example, a fixed area of the screen may be dedicated to a menu, orthe user may explicitly control the positions of permanent menus ortemporary pop-up menus, or the positions and sizes of windows. However,hard-wired or direct-manipulation control becomes problematic whenapplied to dynamic scenes that include autonomous objects and tohead-tracked displays. In this type of situation, continual andunpredictable changes in object geometry or viewing specification resultin continual changes in the spatial and visibility relationships amongthe projections on the view plane. In these cases, view-managementdecisions must be made on the fly if they are to take dynamic changesinto account.

[0006] Augmented reality applications are especially challenging in thisregard. Virtual and physical objects reside in the same 3D space andthere may be no way to control the behavior of many of the physicalobjects. For example, the view through an optical see-through head-worndisplay includes all the physical objects that occupy the user's fieldof view in addition to the virtual objects being displayed. In thiscase, the portion of the field of view that can be augmented may berelatively small.

[0007] It would be desirable to manage the virtual space such thatobjects could be added or moved in a controlled manner. For example, itwould be desirable if virtual annotations could be added to the virtualspace and interspersed among the objects they describe and reconfiguredautomatically and understandably to take into account changes in theobjects themselves and how they are viewed.

SUMMARY OF THE INVENTION

[0008] A method for managing a display space for a 3D environmentincludes the steps of determining the visible surfaces of at least onefirst object in a 3D environment and representing the visible surfacesof the at least one first object as visible space in a view-planerepresentation. A plurality of controllable objects to be placed in thescene are defined and for each controllable object, the available spacein the view-plane representation is determined. The placement of eachcontrollable object in the 3D scene is then determined in accordancewith at least one placement parameter and the available space for thecontrollable object in the view-plane representation.

[0009] Also in accordance with the present invention is a method ofannotating scene objects in a frame of a 3D environment. The methodincludes determining the visible surfaces of at least one scene objectin a 3D environment and representing the visible surfaces as visiblespace in a view-plane representation. At least one annotation objectassociated with at least one scene object to be placed in the scene isdefined. If the annotation object fits within the visible surfaces ofthe associated scene object, then the annotation object is placed withinthe extents of the visible surfaces of the associated scene object. Theplacement of annotation objects which cannot be placed within theextents of the visible surface of the associated scene object in the 3Dscene is determined in accordance with at least one placement parameterand the visible space of the view-plane representation. The annotationobjects can then be displayed in the 3D environment in accordance withthe determined placement.

BRIEF DESCRIPTION OF THE DRAWING

[0010] Further objects, features and advantages of the invention willbecome apparent from the following detailed description taken inconjunction with the accompanying figures showing illustrativeembodiments of the invention, in which:

[0011]FIG. 1 is a flow chart illustrating an overview of a method of 3Dview management in accordance with the present invention;

[0012]FIG. 2 is a top plan view of a scene having two scene objects tobe displayed in a 3D space;

[0013]FIG. 3 is a view of the visible surfaces of the two scene objectsfrom a first view point;

[0014]FIG. 4 is a view of the visible surfaces of the two scene objectsfrom a second view point;

[0015]FIG. 5 is a flow chart illustrating a method of representingvisible surfaces in a 3D environment in a 2D view-plane representation;

[0016]FIG. 6 is a pictorial diagram illustrating the use of axis-alignedrectangles to represent the approximate extents of the visible surfacesin the 3D environment;

[0017]FIG. 7 is a pictorial diagram of the 2D view-plane representationof visible surfaces for the scene depicted in FIG. 4;

[0018]FIG. 8 is a pictorial diagram of the view-plane representation ofvisible surfaces for the scene depicted in FIG. 4 for a case where abinary space partitioning tree operation results in a splitting of thevisible space rectangles representing a scene object;

[0019]FIG. 9 is a flow chart illustrating a method of placingcontrollable objects in a suitable position in a 3D environment based onthe 2D view-plane representation;

[0020]FIG. 10 is a flow chart illustrating an alternate method of 3Dview management in accordance with the present invention;

[0021]FIG. 11 is a pictorial diagram of the view-plane representation ofvisible surfaces for the scene depicted in FIG. 4 and furtherillustrating the largest non-occupied space axis aligned rectangleswhich are defined on the view-plane representation;

[0022]FIG. 12 is a pictorial diagram illustrating a method of placingcontrollable objects in a suitable position in the 3D environment ofFIG. 4 based on the 2D view-plane representation; and

[0023]FIG. 13 is a state diagram illustrating a method for improvingframe to frame temporal continuity in the placement of objects in a 3Denvironment.

[0024] Throughout the figures, the same reference numerals andcharacters, unless otherwise stated, are used to denote like features,elements, components or portions of the illustrated embodiments.Moreover, while the subject invention will now be described in detailwith reference to the figures, it is done so in connection with theillustrative embodiments. It is intended that changes and modificationscan be made to the described embodiments without departing from the truescope and spirit of the subject invention as defined by the appendedclaims.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0025]FIG. 1 is a flow chart illustrating an overview of a method of 3Dview management for a current frame of a 3D scene in accordance with thepresent invention. A 3D image of a frame of a scene or object is to begenerated in a conventional manner. The 3D image can be displayed in anumber of ways which are well known in the art, such as a conventional2D display device, such as a CRT or LCD screen, stationary,non-stationary, handheld, or head-worn displays for virtual reality oraugmented reality applications, including head tracked display devices,and the like. In 3D rendering, objects having defined properties ofsize, position, opacity, and the like are placed in the 3D virtual spaceand are rendered in perspective from a particular view point. Based onthe view point of the user and the properties of the objects in the 3Dvirtual space, referred to herein as scene objects, certain surfaces ofthe scene objects will be visible, while others are occluded by otherscene objects. Objects whose position and geometry are fixed withrespect to the scene will be referred to herein as non-controllablescene objects. Objects whose position and/or size or other visibilityconstraint can be altered will be referred to herein as controllableobjects.

[0026] Controllable objects are associated with a variety of propertiesand placement constraints that will determine if and where the objectswill be placed in the 3D space. The object properties can includeminimum and maximum object size, minimum font size, aspect ratio andother constraints on how the object can appear in the space. Theplacement constraints generally relate to where the object can beplaced. Placement constraints can refer to the image space in general,to non-controllable scene objects, and to other controllable objects.The placement constraints can be a single constraint, such as “alwaysplace A centered within B” or a more flexible set of hierarchical ruleswhich determine placement based on a prioritized rule set. For example arule set for placing controllable object A with respect to object Bcould be, “if B is large enough to accept A with minimum font=8 point,place A within B, else place A above B without overlapping any otherscene object.” These two examples are not exhaustive and merely serve toillustrate the concept of placement parameters for controllable objects.

[0027] Based on the particular application, the object properties andplacement constraints (placement parameters) will be defined for eachcontrollable object (step 105). In certain applications, controllableobjects can be grouped by object type, such as labels, which may have acommon set, or subset, of placement constraints.

[0028] For a current frame of the 3D image space, the set of visiblesurfaces of objects placed in the view space from a selected view pointwill be determined (step 110). The visible surfaces of thenon-controllable scene objects are projected into a 2D representation ofthe scene, referred to as a view-plane representation (step 115). Basedon the view-plane representation and the properties and placementconstraints of the controllable objects, such as labels, annotations,user controlled icons, and the like, the available space for eachcontrollable object is determined (step 120). As used herein, the termavailable space refers to space in the view-plane which satisfies theproperties and placement constraints of the particular controllableobject. Once the available space for the controllable objects isdetermined, the placement of the controllable objects is determined(step 125). From the determined placement in the view-plane, thecontrollable objects can be placed in the 3D space (step 130).

[0029] The controllable object properties can include a placementpriority which determines the order in which the controllable objectswill be placed. The placement constraints can relate to bothnon-controllable scene objects as well as other controllable objects. Asa result, it can be desirable for the available space for thecontrollable objects to be determined in priority order, determining theplacement of higher priority objects first and determining the availablespace for subsequent controllable objects in consideration of suchplacement. For example, if lower priority objects are not allowed tooverlap higher priority controllable objects, previously non-occupiedspace now occupied by a previously placed higher priority object wouldno longer be available space for the lower priority object. Such a“greedy algorithm” can be performed by repeating steps 115, 120 and 125for each controllable object, in the order of priority. Preferably, onlythose portions of the view-plane representation affected by theplacement of an object are recalculated when steps 115, 120 and 125 arerepeated.

[0030] The available space for a controllable object is determined bythat object's properties and placement constraints. However, in somecases, objects can be grouped by an object type, such as label, whichhave common properties and placement constraints. In certain cases,non-controllable objects will include placement constraints that preventoverlap with the visible surfaces of the scene objects. In such cases,in addition to having a data structure which defines the visible spacein the view-plane, it can be efficient to maintain a second datastructure describing the space in the view-plane which is not occupiedby objects of interest. Although such areas in the 3D scene are notgenerally truly empty, these regions can collectively be referred to asnon-occupied space areas and can be considered available for certaintypes of controllable objects. For example, it may be determined thatany type of controllable object can be placed in regions of the 3D imagespace which are occupied by grass. In this case, any visible regions inthe 3D image space identified as grass regions can be considerednon-occupied space.

[0031]FIG. 10 illustrates an alternate method in accordance with thepresent invention for determining the placement of objects in a 3D spacewhich represents both the visible space and non-occupied space of thecurrent frame in a view plane representation. Referring to the flowchart of FIG. 10, steps 105, 110 and 115 are substantially similar tothat described above with respect to FIG. 1. Following step 115, thenon-occupied space is represented as the set of largest axis-alignednon-occupied space rectangles defined by the extents of the view-planeand the visible surfaces in the image space (step 1020). For each objectin turn in priority order, the available space on the view plane will bedetermined (step 120). For those objects which have placementconstraints that result in placement in the non-occupied space, the setof non-occupied space rectangles can be queried to determine thenon-occupied space rectangle which best satisfies the placementconstraints for the controllable object. FIG. 10 illustrates a greedyalgorithm wherein the objects are placed (step 1030) and if there areadditional objects (step 1040) to be placed the flow returns to step 115where the view-plane representation is updated prior to determining theavailable space for the next controllable object. When all of thecontrollable objects have been placed in the view-plane, thecontrollable objects can be placed in the 3D space, such as bydisplaying such objects in the scene.

[0032] The present invention can be further described by way of example.FIG. 2 is a top plan view illustrating an example of a virtual space 200which has two objects, scene object 1 205 and scene object 2 210. Ifthese scene objects have a position and geometry which is fixed, such asbuildings or other fixed scene objects, these objects will be considerednon-controllable scene objects. Not withstanding the designation ofobjects as “non-controllable,” it will be appreciated that certainparameters of the object which do not affect the geometry or position ofthe object may be altered. While the objects can be viewed from anylocation in the virtual space 200, two selected view points 215, 220 areshown as examples. The scene objects can be any arbitrary objects to beviewed or manipulated in the 3D space, such as buildings as part of avirtual map or tour.

[0033]FIG. 3 illustrates the visible surface 305 of scene object 1 andscene object 2 from the first view point 215, which is taken from adirect front view of scene object 1 205. In this view, scene object 2210 is completely occluded by scene object 1 and does not present anyvisible surfaces. In this simple example, the view plane representationof the visible surfaces in the current view could be represented by asingle axis aligned rectangle 305 projected into the 2D view-plane. Asused herein, an axis aligned rectangle is a rectangle whose horizontalaxis and vertical axis are aligned with a respective horizontal axis andvertical axis of the view-plane representation of the environment. Whilenon-axis aligned rectangles, and other geometric entities can be used toapproximate the visible surface areas, it will be appreciated that suchless constrained geometries will increase processing time andcomplexity.

[0034]FIG. 4 illustrates a more generalized example where scene object 1205 and scene object 2 210 are observed from the second view point 220,which is taken from a front right view of scene object 1 205. Asillustrated in FIG. 4, in addition to the front surface 405 of sceneobject 1 205, this second view point also exposes the right side 410 andtop surface 415 of scene object 1 and also exposes a portion of thefront surface 420 of scene object 2 as well as the right side 425 andtop surface 430 of this object.

[0035]FIG. 5 is a flow chart illustrating an example of a method ofrepresenting visible surfaces in a 3D environment in a 2D view-planerepresentation. A number of approaches can be used to determine whichsurfaces of the objects in the scene are visible from a particular viewpoint. One such method includes the use of a binary space partitioningtree. The visible surfaces can also be determined by using a Z-bufferalgorithm. The Z-buffer algorithm can operate at the same resolution asthe rendered scene or can be performed at a lower resolution than therendered scene. In connection with the use of the binary spacepartitioning tree, in step 500, the extents of the scene objects aredetermined for the current scene view. In one embodiment, which isparticularly useful for real-time operation, the extents of the sceneobjects can be estimated by 2D axis aligned rectangles which fullyencompass the extents of the visible surfaces of the individual sceneobjects. The extents of the scene objects are estimated and projectedinto the 2D view plane representation. Referring again to FIG. 5, instep 510, the axis aligned rectangles defined around the extents of eachobject are projected into the 2D view-plane representation of the scenein visibility order. As will be described below in further detail, useof the BSP tree can result in objects being partitioned into adjacentrectangles on the view plane. Therefore, a step of combining adjacentaxis aligned rectangles of the same object on the view-plane isperformed (step 520). The set of largest visible space rectangles on theview-plane can then be determined (step 530). If desired, the set oflargest non-occupied space rectangles on the view plane can also bedetermined (step 540).

[0036] The use of the BSP tree will now be described in greater detail.Visible-surface determination can be performed by sorting the uprightextents of the scene objects' projections in visibility order, such asfront to back with respect to the view point. In the present case,visible-surface determination for interactive rendering is accomplishedindependently of the process for view-plane space determination,preferably by the use of specialized graphics hardware. A Binary SpacePartitioning (BSP) Tree algorithm such as is described in the articleentitled “Set Operations on Polyhedra Using Binary Space PartitioningTrees,” by W. Thibault et al, Computer Graphics, 21(4), pp. 153-162 July1987 (Proc. SIGGRAPH '87), which is hereby incorporated by reference inits entirety, can be used to efficiently produce the visibility orderfor an arbitrary projection of a scene. A BSP tree is a binary treewhose nodes typically represent actual polygons (or polygon fragments)in the 3D scene. Because in the present case it is desirable todetermine the visibility order for objects, rather than for the polygonsof which they are composed, BSP tree nodes in the present invention aregenerally defined by planes that separate objects, rather than planesthat embed objects' polygons. The partitioning planes can be chosen byusing the heuristics described by Thibault et al. in the articlereferenced above. Although BSP trees are often used for purely staticscenes, dynamic objects can be handled efficiently by adding theseobjects to the tree last and removing and adding them each time theobjects move.

[0037] Referring to FIG. 6, the scene of FIG. 4 is illustrated with theextents of the visible surfaces of scene object 1 205 defined by a firstaxis-aligned rectangle 605. Scene object 2 205 is partially occluded byscene object 1. The visible surfaces of scene object 2 205 are definedby two axis aligned rectangles 610, 615. Axis aligned rectangle 610encompasses the right-hand surface 425, the exposed portion of the frontsurface 420 and a portion of the top surface 430 of scene object 2. Athird axis aligned rectangle 615 encompasses the visible top surface 430and a portion of the right-hand surface 425 of scene object 2. FIG. 7illustrates the projection of the three axis aligned rectangles 605, 610and 615 into the 2D view-plane representation.

[0038] When visible surface determination is performed using a BSP treealgorithm, the process of generating the view-plane representation canbe performed by traversing the BSP tree in front-to-back order relativeto the view point to find the visible portions of the scene's objects.The 2D space representation is projected to determine approximations ofthe visible space and non-occupied space portions of the view plane. Foreach node obtained from the BSP tree in front-to-back order, the newnode's upright extent is intersected with the members of the currentlist of largest non-occupied space rectangles. This can be performedefficiently by maintaining the set of largest non-occupied spacerectangles in a 2D interval tree, such as described by H. Samet in thetext “The Design and Analysis of Spatial Data Structures,”Addison-Wesley, Reading, Mass., 1990, or other suitable data structure,to allow an efficient window query to determine the members thatactually intersect the extent. The intersection yields a set ofrectangles, some of which may be wholly contained within others. Thoserectangles which are subsumed in other rectangles are eliminated,resulting in a set of largest rectangles whose union is the visibleportion of the new node.

[0039] During the process of determining the visible surfaces in thescene, scene objects may be intersected by projection planes of thebinary space partitioning tree and, as a result, represented on theview-plane by two or more visible surface regions which are adjacent toone another. As a result, some objects are represented by a single BSPtree node, while others are split across nodes. (Objects may be splitduring BSP tree construction, or split prior to BSP tree construction tobetter approximate a large object by the upright extents of a set ofsmaller objects.) Therefore, it is preferable for a node's visiblerectangles to be coalesced with those of all previously processed nodesfrom the same object to create the list of largest rectangles in theunion of the nodes.

[0040] For example, it is possible for a partitioning plane for thebinary space partitioning tree to be placed adjacent to the rightsurface 410 of scene object 1 205. In this case, instead of the visiblesurfaces of scene object 2 being represented as two overlappingrectangles 610, 615, scene object 2 would be represented in separatenodes of the BSP tree as rectangle 815 and adjacent rectangle 610, asillustrated in FIG. 8. Because of the possibility that objects can bepartitioned and represented in separate nodes of the BSP tree, it isdesirable to have an additional operation wherein the rectanglesrepresenting the extents of the objects in the 2D view-planerepresentation are coalesced into largest axis aligned rectangles foreach object (FIG. 5, step 520). In the case of FIG. 8, the merging ofvisible space rectangles 815 and 610 would yield the two largest visiblespace rectangles 615, 610, previously illustrated in FIG. 6.

[0041] Returning to FIG. 5, following the coalescing of the set oflargest visible space rectangles on the view plane can be determined(step 530) and the set of largest non-occupied space rectangles can thenbe determined (step 540). The process for determining the set of largestnon-occupied space rectangles can be performed in a manner similar tothat described in International patent application PCT/US01/13167,entitled SYSTEM AND METHOD FOR DYNAMIC SPACE MANAGEMENT OF A DISPLAYSPACE, published on Nov. 1, 2001 under publication number WO 01/82279A3, which is hereby incorporated by reference in its entirety.

[0042] As discussed above, the visible surface of objects can beestimated by largest rectangles which encompass the extents of theobject's visible surfaces. In certain cases, this may result in aninefficient allocation of the view plane. In order to more efficientlyallocate the view plane, the extents of the visible surfaces of anobject can be represented by a set of multiple, smaller rectangles whichmore closely follow the contours of the object. For example, consider anobject whose visible extents is shaped like the letter T. This object,when estimated by a single rectangle will be allocated far more space onthe view plane than necessary. However, if the extents of the object'svertical portion is represented by a first rectangle and the extents ofthe horizontal portion represented by as second rectangle, it will beappreciated that the view plane now more efficiently represents thevisible space of the object.

[0043] As noted above, empty space is generally not truly empty, but caninclude regions of the scene which have been deemed as available for theplacement of controllable objects. For example, referring to FIG. 4, thescene may include a horizon line 440. Above the horizon line 440 can bea region 450 rendered as sky and below horizon line 440 can be a region460 rendered as foreground, such as grass. While regions 450 and 460 arenot empty, these regions may, by definition by the user, be consideredas available space in which controllable objects may be placed. In otherwords, the object constraints for regions 450 and 460 do not preventhaving these regions blocked or occluded by controllable objects placedin the scene. In such a case, regions 450 and 460 can be groupedtogether, along with any empty space regions, in defining a set ofnon-occupied space rectangles in the view-plane representation. Thisprocess is illustrated in graphical form in FIG. 11.

[0044] The 2D view-plane representation of FIG. 11 can be represented incomputer readable media as a first data structure of visible rectanglesand a second data structure of non-occupied space rectangles. Using thisview-plane representation, the available space for the placement ofcontrollable objects, such as labels, annotations, user interfacecontrol boxes, text boxes into the 3D scene can be determined in amanner which enhances the scene and complies with the placementconstraints of the scene objects and controllable objects.

[0045] A common form of controlled object is an annotation whichprovides information about an object in the 3D scene, such as a labelwhich identifies a scene object in the 3D scene. In order to bemeaningful, such annotations should be uniquely associated with theobject. One preferred way of associating an annotation with an object isto place the annotation entirely within the visible space of the objector, alternatively, placed in closely located available space with aleader line connecting the annotation to the object. In addition, inmany applications, such as labeling, it is undesirable for theannotations, once placed, to overlap each other.

[0046]FIG. 9 is a flow chart that illustrates an example of the presentinvention in the form of a process of placing controllable objects, suchas annotations, within the 3D scene in accordance with the 2D viewplane. A controllable object can be defined by the user or predefined bythe system's GUI. Controllable objects can be defined by properties andconstraints such as total size, width, height, aspect ratio, font size,opacity, priority and the like. Thus, in step 900, the parameters of thecontrollable objects, such as annotations are defined. If more than onecontrollable object is to be placed, a first controllable object isselected, such as in a priority order (step 905). Controllable objects(cont. object) can have placement constraints which are associated witha particular scene object. This determination is made in step 910. Inthe case of labels, these controllable objects are generally associatedwith a particular scene object and have a size which is largelydetermined by the label's text string, font type, and minimum font size.In the case of controllable objects which are associated with aparticular scene object, it can be desirable to place the controllableobject within the visible rectangle of the scene object. Thus, such acontrollable object would have a placement constraint defining suchpreferred placement. To do this, the controllable object must fitentirely within at least one visible rectangle associated with the sceneobject (step 920). If this condition is satisfied, the controllableobject is placed within the confines of the scene object and, ifadditional objects are to be placed, the next controllable object isselected (step 940) and flow returns to step 910. To avoid overlap ofthe controllable objects, the placement of the controllable objects inthis example is generally determined in accordance with the priorityassigned to the controllable objects or the scene objects with which thecontrollable objects are associated.

[0047] If in step 920, it is determined that the controllable objectdoes not fit within the extents of a visible rectangle of the associatedscene object, the controllable object will be placed in a suitableavailable space rectangle, generally near the associated scene object,in accordance with placement constraints for the object. The view-planerepresentation is queried to determine the set of available spacerectangles that are large enough to receive the controllable object(step 950). In the event that there is more than one acceptableavailable space rectangle, one or more placement parameters associatedwith the object can be used to select one available space rectangle fromthe set. For example, rules can include relationships such as: closestto, smallest, adjacent, above, below, left of, right of, and the like.Such rules can be used alone or in combination in selecting theappropriate available space rectangle. Once an available space rectangleis selected, the controllable object is placed within the selectedavailable space rectangle in a size an position consistent with theplacement constraints for the object (step 960).

[0048] In order to avoid overlap or occlusion of the newly placedcontrollable object, the space occupied by the controllable object isdesignated as visible space and the available space in the view plane isrecalculated (step 970). The next object is selected (step 940) and theprocess is repeated for each controllable object to be placed. If thecontrollable object will not fit within any visible space rectangle orsuitable available space rectangle, the controllable object will not beplaced unless a rule giving this object priority is provided.

[0049] If in step 910, it is determined that a controllable object to beplaced is not associated with a scene object, placement for the objectwill be made in accordance with placement rules associated with theobject and the scene. For example, a pop-up text box for user notes mayhave an associated rule that places this object in the uppermostavailable space rectangle large enough to receive it. Scene-based rulesmay also be used, such as maintaining a central region of the scene freeof controllable objects, such as may be desirable when using a seethrough head mounted display unit to view the 3D environmentsimultaneously with the surrounding physical environment and other usersin a collaborative setting. In such a case, the controllable object isplaced in accordance with the placement parameters (step 985) and thenon-occupied space in the 2D view-plane is recalculated (step 990). Thenext object is selected (step 940) and the process repeats for the nextcontrollable object with flow returning to step 910.

[0050]FIG. 12 is a pictorial example illustrating the placement of twocontrollable objects, such as labels, associated with scene objects 205,210 from FIG. 4. Controllable object 1 is associated with scene object 1205. This controllable object fits within the visible rectangle 605(FIG. 6) of scene object 1 and is placed therein. Controllable object 2is associated with scene object 2 210. Controllable object 2 does notfit within either of the two visible space rectangles 610, 615 for sceneobject 2. Therefore, controllable object 2 is placed in an availablespace rectangle for this object in proximity to scene object 2. Asillustrated in FIG. 10, a number of available space rectangles could beselected to receive controllable object 2. The placement shown in FIG.11, within available space rectangle 1105, is consistent with rules“below” and “smallest.” Since the controllable object is placed outsidethe confines of the visible rectangle of the associated scene object, aleader line can be provided between the scene object and thecontrollable object. A leader line can also be placed in accordance withplacement constraints. For example, a leader line can originate at thecenter of the label and can terminate at the center of the largestvisible space rectangle of the object it refers to. Based on constraintsof the leader line, the portions of the leader line overlapping thelabel or object need not be visible.

[0051] As is evident from FIGS. 3 and 4, as the view point changes, thevisible surfaces of the scene objects change, and therefore, theavailable space representation in the view plane also changes. Thus, foreach frame of the 3D scene, the process of determining the placement ofobjects is applied. As the projection of the scene changes, thecontrollable objects within the scene need to be repositioned inaccordance with the visible space and available space of the currentframe. It is desirable to avoid large transitory jumps of objects withinthe space as the scene changes. Therefore, in the event that because ofa scene change, a significant change in placement of a controllableobject is required, this should be performed in a controlled andcontinuous manner. There are several methods which can be used toimprove the temporal continuity of the display as the scene changes,such as state hysteresis, positional stability and interpolation.

[0052] To avoid having a controllable object repeatedly jump between twopositions as the scene undergoes minor changes in view point from oneframe to the next, it is desirable to include state hysteresis in thepositioning of the controllable objects. There are several situations inwhich objects may change state, resulting in a discrete visual change,such as from an internal label to external one, or from being displayedto not being displayed. Some GUIs use three constraints on the size ofan object being displayed: minimum size, maximum size, and preferredsize (e.g., Java 2D). As shown in the state diagram of FIG. 13 thisapproach can be extended by modifying the definition of minimum size bydefining both an absolute minimum size (absMin) and a sustained minimumsize (min) to provide state hysteresis and avoid oscillation betweenstates at size boundary conditions.

[0053] In the present state hysteresis analysis, the position of anobject in the previous frame is compared to an ideal position for theobject in the current frame. If these two positions are not the same, atimer is initiated. If at the end of the timer interval, the currentposition of the object is not the ideal position, the object is thenmoved to the new ideal position. This may result in a momentarypositioning which is not ideal, but avoids the undesirable jumping orflickering of the controllable objects within the image.

[0054] Referring to FIG. 13, an object's visual size is can beguaranteed to be the sustained minimum size at least once within asettable time interval n. The system will display the object only whenthere is enough space for the sustained minimum size, and removes theobject when it is below the absolute minimum size. Furthermore, if theobject is already being displayed and there is only enough space for anobject within the absolute and sustained minimum sizes for time>n, theobject is removed. Otherwise, the object is displayed at the largestpossible size no greater than its maximum size. The state diagram ofFIG. 13 employs similar tests to make an additional distinction betweendrawing an object inside an associated object as compared to outsideanother object, which can be used for displaying area feature labels.The values selected for n and the difference between the absoluteminimum and sustained minimum sizes help avoid visual discontinuities.

[0055] In addition to state hysteresis, it is desirable to place objectsbeing laid out in the 3D scene in roughly the same position relative toan associated object or to a screen position if the available object isscreen-stabilized as in the previous frame. This is referred to aspositional stability. For an object L being placed relative to object A,two possible layouts can be computed: the best possible layoutindependent of the previous layout, and the closest possible layout tothe previous layout. For example, when I is an internal label for A, thebest possible layout may be to use the visible space in A that cancontain the largest allowable version of L. To determine the closestpossible layout to the previous layout, the position of L's centroid iscomputed in the previous frame relative to A's unclipped width andheight in the previous frame. These proportions can then be used tocompute from A's unclipped width and height in the current frame, apredicted position L^(C) for L's centroid. Next, the best and closestpossible layouts are compared. If they are the same, then this layout isused. If the layouts are different, a timer is initiated and if the bestand closest fail to coincide after a set amount of time, the bestposition is selected and the timer is reset.

[0056] A third method of improving temporal continuity is to interpolatebetween certain kinds of discrete changes. For example, to minimize theeffect of discontinuous jumps during the state changes discussed above,L is interpolated from its previous position and scale to its new ones.In changing from internal to external annotations, the object or leaderline can also grow or shrink.

[0057] In placing the controllable object into the 3D scene, it ispreferable to place the controllable object at a depth, e.g.,z-dimension in the 3D scene, which corresponds to the associated sceneobject. In the case of stereo display devices, the present invention canbe implemented by determining the view plane for a single eye, anaverage distance between the two eyes or individually for each eye.

[0058] The present system and methods have been implemented in Java 1.3with Java 3D 1.2.1.01. The software was operated on a 1.4 GHz IntelPentium 4 processor with 512 MB RAM and a SONICBlue FireGL 2 graphicsboard, running Windows 2000. The visible-surface processing performed bythe methods described herein is only used for view-managementoperations. Rendering is accomplished through a separate processingengine, such as Java3D. While performance depends on the complexity ofthe scene, the present system runs at about 10-25 frames per second instereo for an 800×600 resolution Sony LDI-D100B head-worn display (withthe view-plane representation computed for a single eye).

[0059] The present invention can be implemented on various computerplatforms and use various input devices and display devices. Examplesinclude a conventional CRT display of a 3D space, such as a collegecampus, wherein the scene objects represent buildings on the campus andthe controllable objects include labels identifying the building namesand selectable annotations, including text or multimedia supplementsassociated with a particular building. Thus an interactive tour can beprovided in a 3D environment to one or more users in either a stationaryor mobile embodiment More sophisticated examples of the presentinvention include stationary or mobile augmented reality environmentswhere multiple users engage in a collaboration with respect to a 3Denvironment which is presented to each user through see-throughhead-mounted or hand held or stationary display units, such as the SonyLDI-D100B head-worn display. In a collaborative setting, controllableobjects can be further defined as “private” to a particular user and“public” to all or a group of collaborators.

[0060] Although the present invention has been described in connectionwith specific exemplary embodiments, it should be understood thatvarious changes, substitutions and alterations can be made to thedisclosed embodiments without departing from the spirit and scope of theinvention as set forth in the appended claims.

What is claimed is:
 1. A method for managing a display space for a 3Denvironment comprising: determining the visible surfaces of at least onefirst object in a 3D environment and representing the visible surfacesof the at least one first object as visible space in a view-planerepresentation; defining a plurality of controllable objects to beplaced in the scene; for each controllable object, determining theavailable space in the view-plane representation; determining theplacement of each controllable object in the 3D scene in accordance withat least one placement parameter and the available space for thecontrollable object in the view-plane representation.
 2. The method formanaging a display space of claim 1, wherein the visible space for eachobject is represented as rectangles which are axis aligned with respectto the view-plane representation.
 3. The method for managing a displayspace of claim 2, wherein the axis aligned rectangles comprise the setof largest visible space rectangles on the view-plane representation. 4.The method for managing a display space of claim 1, wherein theavailable space for each object is represented as rectangles which areaxis aligned with respect to the view-plane representation.
 5. Themethod for managing a display space of claim 4 wherein the axis alignedrectangles comprise the set of largest available space rectangles on theview-plane representation.
 6. The method for managing a display space ofclaim 1, wherein at least one of the plurality of controllable objectsis associated with one of the at least one first objects and wherein thecontrollable object is placeable within a visible space rectangle of theassociated first object if the visible space rectangle has a size whichcan receive the projection of the controllable object.
 7. The method formanaging a display space of claim 6, wherein the at least onecontrollable object is placeable within an available space rectangleoutside the associated scene object if no visible space rectangle of theassociated scene object has a size that can receive the projection ofthe controllable object.
 8. The method for managing a display space ofclaim 1, wherein a set of available space rectangles is determined fromthe view-plane representation and one available space rectangle fromsaid set is selected based on the at least one placement parameter. 9.The method for managing a display space of claim 1, wherein the at leastone placement parameter includes a relationship to the position of theprojection of the controllable object in a previous frame of the 3Dscene.
 10. The method for managing a display space of claim 9, whereinthe relationship to the position of the projection of the controllableobject in the previous frame includes temporal state hysteresis.
 11. Themethod for managing a display space of claim 9, wherein the relationshipto the position of the controllable object in the previous frameincludes positional stability with respect to an associated object. 12.The method for managing a display space of claim 9, wherein therelationship to the position of the controllable object in the previousframe includes interpolating between a last frame position and a currentframe position.
 13. The method for managing a display space of claim 1,wherein the first object is a non-controllable object.
 14. The methodfor managing a display space of claim 1, wherein the first object is acontrollable object.
 15. The method for managing a display space ofclaim 1, wherein the controllable objects are displayed in a 3Denvironment in accordance with the determined placement.
 16. The methodfor managing a display space of claim 15, wherein the controllableobjects are displayed using a head-tracked display device.
 17. A methodof annotating scene objects in a frame of a 3D environment comprising:determining the visible surfaces of at least one scene objects in a 3Denvironment and representing the visible surfaces as visible space in aview-plane representation; defining at least one annotation objectassociated with at least one scene object to be placed in the scene; ifthe annotation object fits within the visible surfaces of the associatedscene object, then place the annotation object within the extents of thevisible surfaces of the associated scene object; determining theplacement of annotation objects which cannot be placed within theextents of the visible surface of the associated scene object in the 3Dscene in accordance with at least one placement parameter and thevisible space of the view-plane representation; displaying theannotation objects in the 3D environment in accordance with theplacement.
 18. The method for annotating scene objects of claim 17,wherein the visible space is represented as rectangles which are axisaligned with respect to the view-plane representation.
 19. The methodfor annotating scene objects of claim 17, wherein the available space isrepresented as rectangles which are axis aligned with respect to theview-plane representation.
 20. The method for annotating scene objectsof claim 17, wherein a set of available space rectangles proximate theassociated scene object is determined from the view-plane representationand one rectangle from said set is selected based on at least oneplacement rule.
 21. The method for annotating scene objects of claim 17,wherein the at least one placement rule is related to the position ofthe annotation object in a previous frame of the 3D scene.
 22. A methodfor managing a display space for a 3D environment comprising:determining the visible surfaces of a plurality of scene objects in a 3Denvironment and representing the visible surfaces of the plurality ofscene objects as visible space in a view-plane representation; definingat least one of controllable object to be placed in the scene; for eachcontrollable object, determining the available space in the view-planerepresentation; determining the placement of each controllable object inthe 3D scene in accordance with at least one placement parameter and theavailable space for the controllable object in the view-planerepresentation.
 23. The method for managing a display space of claim 22,wherein the visible space for each object is represented as rectangleswhich are axis aligned with respect to the view-plane representation.24. The method for managing a display space of claim 22, wherein theavailable space for each object is represented as rectangles which areaxis aligned with respect to the view-plane representation.
 25. Themethod for managing a display space of claim 22, wherein at least one ofthe plurality of controllable objects is associated with one of the atleast one first objects and wherein the controllable object is placeablewithin a visible space rectangle of the associated first object if thevisible space rectangle has a size which can receive the projection ofthe controllable object.
 26. The method for managing a display space ofclaim 25, wherein the at least one controllable object is placeablewithin an available space rectangle outside the associated scene objectif no visible space rectangle of the associated scene object has a sizethat can receive the projection of the controllable object.
 27. Themethod for managing a display space of claim 22, wherein a set ofavailable space rectangles is determined from the view-planerepresentation and one available space rectangle from said set isselected based on the at least one placement parameter.
 28. The methodfor managing a display space of claim 22, wherein the at least oneplacement parameter includes a relationship to the position of thecontrollable object in a previous frame of the 3D scene.
 29. The methodfor managing a display space of claim 28, wherein the relationship tothe position of the controllable object in the previous frame includestemporal state hysteresis.
 30. The method for managing a display spaceof claim 28, wherein the relationship to the position of thecontrollable object in the previous frame includes positional stabilitywith respect to an associated object.
 31. The method for managing adisplay space of claim 28, wherein the relationship to the position ofthe controllable object in the previous frame includes interpolatingbetween a last frame position and a current frame position.
 32. Themethod for managing a display space of claim 22, wherein thecontrollable objects are displayed in a 3D environment in accordancewith the determined placement.
 33. The method for managing a displayspace of claim 22, wherein the controllable objects are displayed usinga head-tracked display device.
 34. The method for managing a displayspace of claim 22, wherein the controllable objects are displayed usinga stereo display device.
 35. The method for managing a display space ofclaim 22, wherein the visible space for each object is represented asset of largest visible space rectangles which are axis aligned withrespect to the view-plane representation.
 36. The method for managing adisplay space of claim 22, wherein the available space for each objectis represented as a set of largest available space rectangles which areaxis aligned with respect to the view-plane representation.